nvx-rs: move the CLI's sandbox lifecycle onto the store and supervisor - #772

nvx-rs sandbox kept its own runtime records, lock, and spawn, wait, and kill code. This PR moves its managed lifecycle onto the host layers that every host of a sandbox shares, and the CLI keeps only its configuration record and its policy. #675 prepared the move without changing behavior: it added the reader of runtime identities that start records, and shaped start for the supervisor. host::instance follows in #721, stacked on this PR, with aci_edge_sandboxes and the removal of the old host::process helpers.

Commits

1. nvx-rs: move the CLI's sandbox lifecycle onto the store and supervisor: the move, with its docs, tests, and golden fixture. The CLI's lock, runtime record, and process code move together, since the store's lock guards the store's records, in which the supervisor records the OpenVMM that it launches.
2. scripts: carry guest build provenance in setup bundles.
3. nvx-rs: open the store's files without waiting on a FIFO.
4. nvx-rs: start only the root's OpenVMM, and create state directories in the store.
5. nvx-rs: read install metadata without waiting on a FIFO.
6. nvx-rs: refuse a store lock file that is a link.

What changes

• Store. The state directory is the sandbox's host::store. Its lock, sandbox.lock, serializes provision, start, stop, and deprovision; exec holds it only while it finds the running VM, so a long workload still cannot delay a stop. The store's launch journal and its runtime record of format 4 replace the CLI's runtime record of format 1. config.json, in the CLI's formats 1 to 6 with the distro binding of nvx-rs: separate distro verification from launch planning #653, stays the CLI's own record beside them, and start still checks that binding from metadata alone, and plans the launch from it as nvx-rs: plan sandbox launches with the image format and upper #676 does, before it launches OpenVMM.
• Supervisor. start launches OpenVMM with supervisor::launch, commits once the guest answers with the required control features, and otherwise ends OpenVMM as before and abandons the launch. Every other transition finds the VM with supervisor::reattach, as an orchestrator does after it restarts within the same boot: it ends an interrupted start, holds a live OpenVMM through its verified process identity, and reports, without driving it, a VM that another runtime identity started. stop runs the supervisor's graceful step through the control session, and reports a timeout without ever taking the forced step: managed control response timed out, or OpenVMM did not terminate after managed stop.
• Session. Control goes through host::session over the NVXS host connection, as before.
• Runtime identity. start records the runtime identity that ArtifactDir::runtime_identity, from nvx-rs: prepare the CLI's move onto the store and supervisor #675, reads from the install metadata of the artifacts: a release's SHA256SUMS, or a checkout's build provenance. Nothing hashes an artifact. A managed start runs the root's own OpenVMM, which the metadata identifies, and refuses another one that NVX_OPENVMM names.
• Setup bundles. The guest bundle of scripts/setup carries vmlinux.provenance.json and initramfs.provenance.json, which the Windows setup requires and installs, so that a host whose guest artifacts come from a bundle can start managed sandboxes.
• FIFOs. Nothing that a transition reads while the sandbox is locked waits on a FIFO: store::files::open_lock opens sandbox.lock without waiting for a reader, store::files::read_private opens the runtime record, the launch journal, and the capability without waiting for a writer, and host::artifacts reads the install metadata of the runtime identity as the store reads its records. Each refuses a file that is not a regular file before it uses it. The CLI's own lock was opened for reading and writing, which never waited, and the CLI checked that its records were regular files before it read them.
• Lock links. store::files::open_lock refuses a sandbox.lock that is a link, or on Windows any other reparse point, as the CLI's own lock did: a lock taken through a link would hold the file that the link names, which a launch may replace while the lock is held, so that the next transition would lock the replacement. On Windows the lock file is opened as itself and checked through the opened handle; on Unix, O_NOFOLLOW refuses a link.
• Removed: the CLI's lock and its tombstone handoff, its process-identity module (cli/sandbox/identity.rs), the Windows byte-range lock helpers, and the fstat helpers that only they used.
• Starting::child, behind the cli feature, gives the CLI the launched child, so that a failed start ends OpenVMM as it did before abandoning the launch.

Behavior changes

• Stale runtime records. reattach clears the runtime record of an OpenVMM that exited without a stop. The first exec or stop reports the stale state once; then stop reports that the sandbox is not running, start launches it again, where it used to refuse, and deprovision removes it. The crash step of the sandbox-lifecycle scenario follows this. It is the one lifecycle test that changes beyond the format switch: nvx-rs: share the VMM supervisor and the sandbox store #620 expects the lifecycle tests to pass unchanged, but keeping the old refusal would need a record of the CLI's own beside the store's.
• Private state directories. provision refuses an existing state directory that is not private to the current user, as the store does, instead of making it private.
• Each start replaces openvmm.log, which the store creates fresh for each launch, instead of appending to it.
• A managed start refuses an OpenVMM that NVX_OPENVMM names instead of the root's, since no metadata identifies it.

Format switch

Switching to this build needs one stop of every running sandbox, with the build that started it. Nothing is migrated: the store refuses a runtime record of format 1 in every transition, with a message that names the format and asks to stop the sandbox with the build that started it. A stopped sandbox keeps its config.json and starts with this build, but the earlier build's lifecycle.lock stays in its state directory, and deprovision reports it as a file not owned by NVX: deprovision stopped sandboxes with the earlier build, or remove that file. On Windows, a state directory that an earlier build created outside the user's profile may not be private to the user, and the store refuses it: deprovision it with the earlier build, and provision it again. An OpenVMM that an interrupted start of an earlier build left behind has no runtime record, and that build asked for it to be ended by hand, with its control.capability and control.sock: do so before the switch too, since this build finds no VM in that sandbox.

The golden fixture nvx-rs/tests/fixtures/cli.json is updated by hand for the switch, as nvx-rs/README.md asks of deliberate changes: the lock's name, the store's runtime record, the log that each start replaces, and the build provenance in the fixture's root.

Tests

• nvx-rs/tests/cli.rs:
  • a restarted CLI drives a live VM of its runtime identity, and reports one of another identity, naming the parts that differ, without driving it, until the artifacts that started it return;
  • a runtime record of format 1 fails every transition with a message that names its format, and is left as it is;
  • stop reports a guest that never answers and an OpenVMM that outlives the wait, and leaves the VM running;
  • a sandbox whose OpenVMM exited reports it once, and starts again;
  • a managed start refuses an OpenVMM that NVX_OPENVMM names.
• host::store unit tests: a FIFO at sandbox.lock fails at once, with and without a reader, and so does one at the runtime record, the launch journal, or the capability, with and without a writer; and a sandbox.lock that links to the capability fails at once, and leaves the link and the capability as they are. The link test skips on a host that does not let it create a symbolic link.
• host::artifacts unit test: a FIFO at a build provenance or at a release's checksum list fails at once, with and without a writer.
• The fake OpenVMM gains fake_exit_delay_ms, which delays its exit after an acknowledged stop.

Part of #620. Refs #535.

Validation

Run on Windows 11 (MSVC, WHP) and on an Azure Linux 3 host with MSHV. The MSHV host cannot present the built-in profile of its CPU (amd.genoa.v1), so OpenVMM ran there through a wrapper that adds --cpu-profile host.

• At this head (be28d631), on dev at 72e8ea26, which merged nvx-rs: prepare the CLI's move onto the store and supervisor #675. Real VMs ran with nvx-rs built from this branch and the artifacts of v0.1.0-dev.72e8ea2677bd, the release of 72e8ea26, from nvx.py download, whose OpenVMM is 7ff31ed7, the gitlink of dev.
  • python scripts/nvx.py verify.
  • nvx-rs, with NVX_BUNDLE=skip, on Windows and Linux: cargo fmt --all --check; cargo clippy --locked --all-targets --all-features -- -D warnings; cargo clippy --locked --lib --no-default-features -- -D warnings; cargo test --locked --all-features, then cargo test --locked; cargo doc --locked --no-deps --all-features with RUSTDOCFLAGS=-D warnings. On Linux, the other steps of check-nvx-rs: the 1.89 check, the macOS check, the artifact-bundling test, and the three configurations of tests/runtime-adapter. The new link test ran on both, and fails on Windows without the last commit, where the lock holds control.capability.
  • aci_edge_sandboxes: the steps of check-aci-edge-sandboxes on Linux (fmt, clippy, tests, nvx.py test-agent-bundling, docs, the 1.89 check, the macOS check), and its fmt, clippy, tests, and docs on Windows.
  • Each commit builds on its own: cargo check --locked --all-targets --all-features in nvx-rs and aci_edge_sandboxes, on Linux.
  • On Linux, the Specula tests and the pinned shellcheck and shfmt on the scripts that check-quality lists, and the tests of validate-nvx, test_setup_azure.py included.
  • WHP: python scripts/nvx.py test-microvm --scenario managed-lifecycle --scenario sandbox-lifecycle --scenario managed-exec-config, and python scripts/nvx.py test-aci-edge-sandboxes (18 tests).
  • MSHV, under sudo for /dev/mshv: the same three scenarios; and test-aci-edge-sandboxes, where 13 of 18 tests pass and the 5 that write through read-write mappings fail as they do on dev, because the unprivileged workload cannot write the root-owned directories that a root test creates.
• At 0e458e5e, without the last commit, which changes only nvx-rs: the same gates, and also, on Windows, the Python steps of check-quality with the pinned tools of requirements-dev.txt (ruff check, ruff format --check, pyright for Linux and Windows, the Markdown link check), the PowerShell syntax check, the tests of validate-nvx, and the nvx.py --help smoke commands. On Windows, four process-containment tests of test_adversarial.py fail identically on dev without this PR, and its test_close_is_bounded_when_a_surviving_child_holds_executor_stdout fails in some runs, as it does on dev's Python, because the removal of its temporary directory races the exit of the child that it killed. Real VMs: on WHP, the same three scenarios and test-aci-edge-sandboxes (18 tests), with the artifacts of v0.1.0-dev.bdaed9c6a7ca; on MSHV, the same as at this head, with those of v0.1.0-dev.8483e3757469.
• Before the split, nvx-rs: prepare the CLI's move onto the store and supervisor #675's 141a04b4, on dev at e714acbd, held the same changes but the FIFO and link checks of the store and of the install metadata, and passed the same gates on Windows and Linux, and the same real-VM runs on WHP, KVM, and MSHV with the artifacts of v0.1.0-dev.e714acbd47bf.

Left to CI: the full matrix on KVM, MSHV, and WHP.